1. Field of the Invention
The present invention relates to computation systems, and, more particularly, to input encoders for neural network and related distributed memory and computation devices.
2. Description of the Related Art
Attempts to understand the functioning of the human brain have led to various "neural network" models in which large numbers of neurons are interconnected. These models roughly presume each neuron exists in one of two states (quiescent and firing) with the state determined by the states of connected nuerons (if enough connected neurons are firing, then the original neuron should be in the firing state); and the thrust of the models is to perform computations such as pattern recognition with the neural networks.
J. Hopfield, Neural Networks and Physical Systems with Emergent Collective Computational Abilities, 79 Proc. Natl. Acad. Sci. USA 2554 (1982) describes a neural network model with N neurons each of which has the value 0 or 1 (corresponding to quiescent and to firing), so the state of the network is then a N-component vector V=[V.sub.1, V.sub.2, . . . , V.sub.N ] of 0's and 1's which depends upon time. The neuron interconnections are described by a matrix T.sub.i,j defining the influence of the j.sup.th neuron on the i.sup.th neuron. The state of the network evolves in time as follows: for each i the i.sup.th neuron has a fixed threshold .theta..sub.i and readjusts its state V.sub.i randomly in time by setting V.sub.i equal to 0 or 1 depending on whether ##EQU1## is negative or positive. All neurons have the same average rate of readjustment, and the readjustments define a dynamical flow in state space.
With the assumption that T.sub.i,j is symmetric, the potential function ##EQU2## can be used to show that the flow of the network is to local minima of the potential function. Further, with a given set of uncorrelated N-component vectors U.sup.1, U.sup.2, . . . , U.sup.s, a T.sub.i,j can be defined by ##EQU3## and with the thresholds equal to 0, these U.sup.k are the fixed points of the flow and thus stable states of the network. Such a network can act as a content-addressable memory as follows: the memories to be stored in the network are used to construct the U.sup.k and hence T.sub.i,j, so the stored memories are fixed points of the flow. Then a given partial memory is input by using it to define the initial state of the network, and the state will flow usually to the closest fixed point/stable state U.sup.k which is then the memory recalled upon input of the partial memory.
Further analysis and modified network models appear in, for example, J. Hopfield et al, Computing with Neural Circuits: A Model, 233 Science 625 (1986) and J. Hopfield, Neurons with Graded Response Have Collective Computational Properties like Those of Two-State Neurons, 81 Proc. Natl. Acad. Sci. USA 3088 (1984).
D. Ackley et al, A Learning Algorithm for Boltzmann Machines, 9 Cognitive Science 147 (1985) describe neural networks with additional adjustment mechanisms for the neurons which analogize thermal fluctuations; this permits escape from local minima of the potential function. However, this disrupts the flow to fixed points for memory recall of the Hopfield type neural networks.
L. Cooper, A Possible Organization of Animal Memory and Learning, Proc. Nobel Symp. Coll. Prop. Phys. Sys. 252 (Academic, N.Y. 1973) observes that the modelling of neural networks for animal memory and learning has the problem of mapping events in the animal's environment (i.e., sensory output) to signal distributions in the animal's neurons with the fundamental property of preserving closeness or separateness (in some sense not yet completely defined) of the events. That is, with a vector representation of the neural network states, two events as similar as a white cat and a gray cat should map into vectors which are close to parallel while two events as different as the sound of a bell and the sight of food should map into vectors that are close to orthogonal. Note that standard analysis, such as described in Gonzalez and Wintz, Digital Image Processing (Addison-Wesley 1977), does not use neural network computation and does not have this problem; rather, the standard analysis attempts to extract features and categorize by serial number crunching.
Attempts to encode data while preserving closeness include methods such as the thermometer code which maps an integer k to a vector with k leading 1's followed by n-k 0's. This severely limits the capacity of the neural network and usually requires data compression or resolution loss.
Thus the problem of encoding sensor output for neural network input to preserve some sense of closeness is not solved in the known neural networks except in the extreme cases of codes such as the thermometer code that have limited capacity. SUMMARY OF THE INVENTION
The present invention provides encoders and neural network computers in which neural network compatible code vectors are generated recursively by changing random vector components to yield a sequence of code vectors for encoding by a map of a range of inputs to the sequence of code vectors. This encoding preserves closeness of the inputs in terms of Hamming type distance between the image vectors by mapping close inputs to vectors nearby in the sequence. Further, in contrast to closeness preserving codes such as the thermometer code, the inventive encoding provides a large capacity for the neural network. Preferred embodiments include use of neural networks with neurons having only two states, binary vectors, and integer inputs so that the mapping of a range of integers to the sequence of binary vectors is simply indexing.
This encoder solves the problem of preserving closeness for encoding sensor output to nerual network compatible input while retaining a large capacity to avoid data compression or resolution loss.